Reliability of gaze tracking data for left and right eye

ABSTRACT

Circuitry of a gaze/eye tracking system obtains one or more images of a left eye and one or more images a right eye, determines a gaze direction of the left eye based on at least one obtained image of the left eye, determines a gaze direction of the right eye based on at least one obtained image of the right eye, determines a first confidence value based on the one or more obtained images of the left eye, determines a second confidence value based on the one or more obtained images of the right eye, and determines a final gaze direction based at least in part on the first confidence value and the second confidence value. The first and second confidence values represent indications of the reliability of the determined gaze directions of the left eye and the right eye, respectively. Corresponding methods and computer-readable media are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 15/700,878, filed Sep. 11, 2017. Theabove-identified patent application is hereby incorporated by referenceherein in its entirety for all purposes.

BACKGROUND OF THE INVENTION

The present disclosure generally relates to the field of eye tracking.In particular, the present disclosure relates to systems and methods forgenerating and/or using gaze tracking data indicating a gaze directionof an eye.

Several different eye tracking systems are known in the art. Suchsystems may for example be employed to allow a user to indicate alocation at a computer display by looking at that point. The eyetracking system may capture images of the user's face, and then employimage processing to extract key features from the user's face, such as apupil center and glints from illuminators illuminating the user's face.The extracted features may then be employed to determine where at thedisplay the user is looking. Naturally, factors such as accuracy, speed,and reliability/robustness of the eye tracking are desirable to achievea positive user experience. Therefore, several schemes have beenproposed for mitigating the negative effects of different types oferrors or inaccuracies that may occur in eye tracking systems.

For example, US 2010/0328444 A1 (which is incorporated by referenceherein in its entirety) discloses an eye tracker which includes at leastone illuminator for illuminating an eye, at least two cameras forimaging the eye, and a controller. The configuration of theilluminator(s) and cameras is such that, at least one camera is coaxialwith a reference illuminator and at least one camera is non-coaxial witha reference illuminator. The controller is adapted to select one of thecameras to be active to maximize an image quality metric and to avoidobscuring objects. The eye tracker is operable in a dual-camera mode toimprove accuracy. A method and computer-program product for selecting acombination of an active reference illuminator from a number ofreference illuminators, and an active camera from a plurality of camerasare also disclosed.

As another example, US 2014/0362446 A1 (which is incorporated byreference herein in its entirety) discloses a head mountable display(HMD) system. The HMD system comprises an eye position detectorcomprising one or more cameras configured to detect the position of eachof the HMD user's eyes, a dominant eye detector configured to detect adominant eye of the HMD user, and an image generator configured togenerate images for display by the HMD in dependence upon the HMD user'seye positions. The image generator is configured to apply a greaterweight to the detected position of the dominant eye than to the detectedposition of the non-dominant eye. The dominant eye is detected by one ormore of the following: (i) the eye with the wider range of movement,(ii) the eye which reaches a gaze direction appropriate to a displayedstimulus the closest and/or the most quickly, (iii) the eye which holdsa position relating to a displayed stimulus.

As a further example, US 2016/0004303 A1 (which is incorporated byreference herein in its entirety) discloses an eye gaze tracking systemwhich includes a gaze data acquisition system including a plurality oflight sources and a plurality of image sensors. The light sources arearranged to emit light to a head of the user, and the image sensors areconfigured to receive the light. In an embodiment, the system furtherincludes a gaze tracking module including an ocular feature extractionmodule, a point of regard (POR) calculation module and a POR averagingmodule. The ocular feature extraction module is configured to processthe gaze data and to extract ocular features in relation to each imagesensor, and is configured to determine a confidence value associatedwith an accuracy of the parameters for each image sensor. The confidencevalue may for example depend on head pose angle (in yaw, pitch and rollangle), user distance and feature detection reliability. The PORcalculation module is configured to determine a POR from the ocularfeatures for each image sensor. The POR averaging module is configuredto determine an average POR using the confidence values of the POR forthe respective image sensors.

However, it would be desirable to provide further systems and methodsaddressing at least one of the issues described above.

BRIEF SUMMARY OF THE INVENTION

An object of the present disclosure is to address at least one of theissues described above.

According to a first aspect, there is provided an eye tracking systemcomprising a circuitry. The eye tracking system may also be referred toas a gaze tracking system. The circuitry is configured to obtain one ormore images of a left eye of a user and one or more images of right eyeof the user, determine (or compute) a gaze direction of the left eye ofthe user based on at least one obtained image of the left eye, anddetermine (or compute) a gaze direction of the right eye of the userbased on at least one obtained image of the right eye. The circuitry isconfigured to determine (or compute) a first confidence value based onthe one or more obtained images of the left eye. The first confidencevalue represents an indication of the reliability of the determined gazedirection of the left eye. The circuitry is configured to determine (orcompute) a second confidence value based on the one or more obtainedimages of the right eye. The second confidence value represents anindication of the reliability of the determined gaze direction of theright eye. The circuitry is configured to determine (or compute) a finalgaze direction based at least in part on the first confidence value andthe second confidence value. The final gaze direction may also bereferred to as a combined gaze direction.

Since the left and right eyes are located at different positions,optical effects caused by glasses or obscuring elements such as eyelashes or the nose may have different impact on the gaze directionsdetermined for the left and right eyes, even if the same image isemployed for the gaze tracking of both eyes. The different positioningof the two eyes typically also causes the eyes to be directed inslightly differently directions when a user focuses at a certain pointon for example a computer screen. The different positions and/ororientations of the left and right eye may for example cause glints fromilluminators to be located at different parts of the two eyes. Sincedifferent parts of the eyes have different shape, this may cause gazedirections determined based on the glints to be of differentquality/accuracy for the left and right eye. Optical properties of theleft and right eyes themselves may also be different. For example, theshape of cornea may differ between the eyes. The offset between theoptical center of the eye and the position of the fovea may also differbetween the left and right eye. While the differences may be relativelysmall, such differences may cause the gaze direction determined for oneof the eyes to be less reliable than the gaze direction determined forthe other eye.

As described above, several different factors may potentially cause thedetermined gaze direction of one of the eyes to me more reliable thanthe determined gaze direction of the other eye. If a final gazedirection (or a gaze point) is computed via a simple average of thedetermined gaze directions for the left and right eyes, a temporaryerror in the determined gaze direction of one eye may cause large errorsin the resulting final (or combined) gaze direction, even if thedetermined gaze direction of the other eye is very reliable for thatperiod of time. Determining confidence values for the left and righteyes, and determining a final (or combined) gaze direction based on theconfidence values allows such factors to be taken into account forproviding a more accurate final (or combined) gaze direction for theuser. For example, if the determined gaze direction for one eye isdetermined to be unreliable, this gaze direction may be provided lowerweight than the gaze direction determined for the other eye.

The left and right eyes may for example be illuminated by one or moreilluminators, which may for example be light sources such as lightemitting diodes.

The images of the left eye and the right eye may for example have beencaptured by one or more image sensors, such as one or more cameras.

The determined gaze direction of the left eye may for example define anestimated gaze point of the left eye. Similarly, the determined gazedirection of the right eye may for example define an estimated gazepoint of the right eye.

It will be appreciated that the reliability of the determined gazedirection for the left eye may for example be indicated via a positivescale where high confidence values indicate high reliability/confidence,or via a negative (or inverted) scale where high confidence valuesindicate low reliability/confidence. In other words, a high confidencevalue may be employed to indicate high reliability or may be employed toindicate high uncertainty/unreliability. Similarly, it will beappreciated that the reliability of the determined gaze direction forthe right eye may for example be indicated via a positive scale wherehigh confidence values indicate high reliability/confidence, or via anegative (or inverted) scale where high confidence values indicate lowreliability/confidence.

The first confidence value may for example be determined (or computed)based on one or more parameters representing respective factorsindicative of (or affecting) a reliability of the gaze directiondetermined for the left eye. The one or more parameters may for examplebe computed (or determined) based on the one or more obtained images ofthe left eye. The first confidence value may for example be referred toas a first combined reliability parameter, or a combined reliabilityparameter for the left eye.

The second confidence value may for example be determined (or computed)based on one or more parameters representing respective factorsindicative of (or affecting) a reliability of the gaze directiondetermined for the right eye. The one or more parameters may for examplebe computed (or determined) based on the one or more obtained images ofthe right eye. The second confidence value may for example be referredto as a second combined reliability parameter, or a combined reliabilityparameter for the right eye.

According to some embodiments, the circuitry may be configured todetermine the final gaze direction based on the determined gazedirection of the left eye, the determined gaze direction of the righteye, the first confidence value, and the second confidence value.

According to some embodiments, the determination of the final gazedirection may be based at least in part on a weighted combination (orweighted sum) of the determined gaze direction of the left eye and thedetermined gaze direction of the right eye.

In the weighted combination, the determined gaze direction of the lefteye may for example be weighted based on (or by) the first confidencevalue, and the determined gaze direction of the right eye may forexample be weighted based on (or by) the second confidence value.

According to some embodiments, the circuitry may be configured todetermine (or estimate), based on the one or more obtained images of theleft eye, one or more glint positions in a predetermined region of theleft eye, and to determine (or estimate), based on the one or moreobtained images of the right eye, one or more glint positions in apredetermined region of the right eye.

The left and right eyes may for example be illuminated by one or moreilluminators, which may for example be light sources such as one or morelight emitting diodes.

The glints at the left and right eyes may for example be caused byreflection of light from one or more illuminators illuminating the eyes.

According to some embodiments, the circuitry may be configured todetermine (or estimate), based on the one or more images of the lefteye, a position of a pupil of the left eye and one or more glintpositions at the left eye, and to determine the gaze direction of theleft eye based on the one or more glint positions and the position ofthe pupil. The circuitry may be configured to determine (or estimate),based on the one or more images of the right eye, a position of a pupilof the right eye and one or more glint positions at the right eye, andto determine the gaze direction of the right eye based on the one ormore glint positions and the position of the pupil.

According to some embodiments, the predetermined region of the left eye(and analogously for the predetermined region of the right eye) mayinclude a first region that extends from a cornea center to an edge of aspherical region of the cornea, a second region that extends from theedge of the spherical region of the cornea to an edge of the cornea,and/or a third region that is located outside of the edge of the cornea.

The cornea of the eye is typically approximately spherical in a centralregion of the cornea located around the pupil, but deviates more fromthe spherical shape further away from the center of the cornea. Thecentral region of the cornea may therefore be referred to as a sphericalregion, while the region of the cornea outside the spherical region maybe referred to as a non-spherical region.

According to some embodiments, the third region may be located at thesclera.

According to some embodiments, the determination of the first confidencevalue may be based at least in part on a distance between a corneacenter (or a pupil center) of the left eye and one or more glints at theleft eye. The determination of the second confidence value may be basedat least in part on a distance between a cornea center (or a pupilcenter) of the right eye and one or more glints at the right eye.

The cornea of the eye is typically approximately spherical in a centralregion around the pupil, but deviates more from the spherical shapefurther away from the center of the cornea and thus also from the centerof the pupil. The optical properties of the eye may therefore be moredifficult to model in these regions, which makes glints located far outon the cornea (or even as far out as the sclera) less reliable forcomputation of gaze directions.

Determining the first confidence value may for example compriseassociating a first value of the distance (that is, the distance betweenthe one or more glints and the cornea center of the left eye) with alower reliability than a reliability associated with a second value ofthe distance. The first value of the distance may be higher than thesecond value of the distance. In other words, a gaze direction obtainedusing glints located far from the cornea center may be assigned (orattributed) a lower reliability than a reliability assigned to (orattributed to) gaze directions obtained using glints located closer tothe cornea center. In other words, the reliability may decrease when theglint moves further away from the cornea center. It will be appreciatedthat other factors may also affect the overall reliability of thedetermined gaze directions, and that the overall reliability of adetermined gaze direction may therefore increase even if the glint movesaway from the cornea center.

According to some embodiments, the determination of the first confidencevalue may include associating glints located at a first region of theleft eye with a higher reliability than a reliability associated withglints located at a second region of the left eye. The first region ofthe left eye may extend from a cornea center to an edge of a sphericalregion of the cornea. The second region of the left eye may extend fromthe edge of the spherical region of the cornea to an edge of the cornea.In other words, gaze directions obtained using glints located in thefirst region may be assigned (or attributed) higher reliability thatreliability attributed to (or assigned to) gaze directions obtainedusing glints located in the second region. In other words, thereliability of the determined gaze direction may decrease as the glintsmove from the first region to the second region. It will be appreciatedthat other factors may also affect the overall reliability of thedetermined gaze direction, and that the overall reliability maytherefore increase even if the glints move from the first region to thesecond region.

According to some embodiments, the determination of the first confidencevalue may include associating glints located at the second region of theleft eye with a higher reliability than a reliability associated withglints located in a third region of the left eye. The third region ofthe left eye may be located outside of the edge of the cornea. In otherwords, gaze directions obtained using glints located in the secondregion may be assigned (or attributed) higher reliability thatreliability attributed to (or assigned to) gaze directions obtainedusing glints located in the third region. In other words, thereliability of the determined gaze direction may decrease as the glintsmove from the second region to the third region. It will be appreciatedthat other factors may also affect the overall reliability of thedetermined gaze direction, and that the overall reliability couldtherefore increase even if the glints move from the second region to thethird region.

According to some embodiments, the determination of the secondconfidence value may include associating glints located at a firstregion of the right eye with a higher reliability than a reliabilityassociated with glints located at a second region of the right eye. Thefirst region of the right eye may extend from a cornea center to an edgeof a spherical region of the cornea. The second region of the right eyemay extend from the edge of the spherical region of the cornea to anedge of the cornea. In other words, gaze directions obtained usingglints located in the first region may be assigned (or attributed)higher reliability that reliability attributed to (or assigned to) gazedirections obtained using glints located in the second region. In otherwords, the reliability of the determined gaze direction may decrease asthe glints move from the first region to the second region. It will beappreciated that other factors may also affect the overall reliabilityof the determined gaze direction, and that the overall reliability maytherefore increase even if the glints move from the first region to thesecond region.

According to some embodiments, the determination of the secondconfidence value may include associating glints located at the secondregion of the right eye with a higher reliability than a reliabilityassociated with glints located in a third region of the right eye. Thethird region of the right eye may be located outside of the edge of thecornea. In other words, gaze directions obtained using glints located inthe second region may be assigned (or attributed) higher reliabilitythat reliability attributed to (or assigned to) gaze directions obtainedusing glints located in the third region. In other words, thereliability of the determined gaze direction may decrease as the glintsmove from the second region to the third region. It will be appreciatedthat other factors may also affect the overall reliability of thedetermined gaze direction, and that the overall reliability couldtherefore increase even if the glints move from the second region to thethird region.

According to some embodiments, the determination of the first confidencevalue may be based on a position of one or more glints at the left eyerelative to a position of the pupil edge of the left eye. Thedetermination of the second confidence value may be based on a positionof one or more glints at the right eye relative to a position of thepupil edge of the right eye.

The edge of the pupil may be employed for estimating the position andthe size of the pupil. If the glint is located at the edge of the pupilit will impact the pupil edge detection. Fewer detected pupil edgepoints will make pupil position and pupil size determination lessreliable, which may affect the accuracy of the pupil center and pupiledge computation, which may affect the reliability of the determinedgaze direction. If a glint is located at the edge of the pupil in abright pupil image (where the pupil is illuminated so that it is brightin the image) the brightness of the pupil may cause the position of theglint to be incorrectly estimated, making the gaze tracking dataunreliable. In this case the glint located on the pupil edge may impactthe pupil edge detection at the same time as the glint positiondetermination is degraded, both of which may affect the reliability ofthe determined gaze direction.

The determination of the first confidence value (and analogously thedetermination of the second confidence value) may for example includeassociating gaze directions determined based on glints overlapping thepupils edge with lower reliability than a reliability associated withgaze directions determined based on glints not overlapping the pupiledge. In other words, gaze directions obtained using glints overlappingthe pupil edge may be assigned (or attributed) lower reliability thanreliability attributed to (or assigned to) gaze directions obtainedusing glints not overlapping the pupil edge.

According to some embodiments, the circuitry may be configured todetermine a pupil center of the left eye, a pupil center of the righteye, an eyeball center of the left eye and an eyeball center of theright eye.

According to some embodiments, the cornea center may be a position atthe spherical region of the cornea where a virtual line extending fromthe eyeball center through the pupil center intersects the sphericalregion of the cornea.

According to some embodiments, the determination of the first confidencevalue may be based at least in part on a contrast between a pupil and aniris of the left eye. High pupil-iris contrast (such as much brighterpupil than iris, or much brighter iris than pupil) facilitatesdetermination of the pupil center, the pupil position and the pupilsize. If the pupil-iris contrast is too low, it may be difficult toestimate the position and size of the pupil, which may affect thereliability of the determined gaze direction.

The determination of the first confidence value may for example includeassociating a first level of the contrast with a lower reliability thana reliability associated with a second level of the contrast. The firstlevel of the contrast may be lower than the second level of thecontrast. In other words, gaze directions determined based on imageswith high contrast between pupil and iris may be assigned (orattributed) higher reliability than reliability attributed to (orassigned to) gaze directions determined using images with low contrastbetween pupil and iris. In other words, the reliability of thedetermined gaze direction may decrease as the contrast between pupil andiris decreases.

According to some embodiments, the determination of the secondconfidence value may be based at least in part on a contrast between apupil and an iris of the right eye. The determination of the secondconfidence value may for example include associating a first level ofthe contrast (that is, the pupil-iris contrast for the right eye) with alower reliability than a reliability associated with a second level ofthe contrast. The first level of the contrast may be lower than thesecond level of the contrast.

According to some embodiments, the determination of the first confidencevalue may be based at least in part on a first aggregation level ofdetermined gaze positions of the left eye with respect to apredetermined position (or a reference position) at a display apparatus.Similarly, the determination of the second confidence value may be basedat least in part on a second aggregation level of determined gazepositions of the right eye with respect to a predetermined position at adisplay apparatus.

The same display apparatus and predetermined position may for example beemployed for both eyes. Alternatively, different predetermined positionsand/or display apparatus may for example be employed for the left andright eyes.

It will be appreciated that the aggregation level is a measure of howclose together at the display apparatus the determined gaze positionsare located. The aggregation level may also be regarded as a measure ofstatistical variability, or the size of random errors. The aggregationlevel may also be regarded as a measure of precision of the gazetracking.

High statistical variability (or large random errors) may indicate thatthe determined gaze direction is less reliable. Therefore, thedetermination of the first and second confidence values may for examplecomprise associating a first level of statistical variability with alower reliability than a reliability associated with a second level ofstatistical variability. The first level or statistical variability maybe higher than the second level of statistical variability.

The level of aggregation (or the level of statistical variability) mayfor example be monitored during gaze tracking, and/or may be determinedduring calibration of the gaze tracking system.

According to some embodiments, the determination of the first confidencevalue may be based at least in part on a first distance between apredetermined position at a display apparatus and an average ofdetermined gaze positions (for example at the display apparatus) of theleft eye. The determination of the second confidence value may be basedat least in part on a second distance between a predetermined positionat a display apparatus and an average of determined gaze positions (forexample at the display apparatus) of the right eye.

The same display apparatus and predetermined position may for example beemployed for both eyes. Alternatively, different predetermined positionsand/or display apparatus may for example be employed for the left andright eyes.

The first and second distances between a predetermined position and anaverage of determined gaze positions may for example be regarded as ameasure of statistical bias, or of the size of systematic errors, of theaccuracy of the gaze tracking.

Large statistical bias may indicate that the determined gaze directionsare less reliable. Therefore, the determination of the first and secondconfidence values may for example comprise associating a first level ofstatistical bias with a lower reliability than a reliability associatedwith a second level of statistical bias. The first level or statisticalbias may be higher than the second level of statistical bias. In otherwords, gaze directions obtained when the level of statistical bias islow may be assigned (or attributed) a higher reliability thanreliability assigned to (or attributed to) gaze directions obtained whenthe level statistical bias is high. In other words, the reliability maydecrease when the level of statistical bias increases. It will beappreciated that other factors may also affect the reliability of thedetermined gaze direction, and that the overall reliability maytherefore decrease even if the statistical bias decreases.

The first and second distances between a predetermined position and anaverage of determined gaze positions for the left and right eyes,respectively, may for example be monitored during gaze tracking, or maybe determined during a calibration step.

According to some embodiments, the determination of the first confidencevalue may be based at least in part on a glint intensity at the lefteye, a shape of glints at the left eye, and/or a number of glints at theleft eye. Similarly, the determination of the second confidence valuemay be based at least in part on a glint intensity at the right eye, ashape of glints at the right eye, and/or a number of glints at the righteye.

The glints may for example be located at the corneas of the left andright eyes, respectively.

If the intensity of a glint is too high, it may result in a large whitespot in an image, making it difficult to determine the position of thecenter of the glint, which may affect the reliability of the determinedgaze direction for that eye. Glints with high intensity may for examplecause pixels in the image to be saturated so that a large portion of theglint looks just as bright in the image as the center of the glint. Itmay also be difficult to accurately determine the position of the glintif the glint has too low intensity. Therefore, glints with too high ortoo low intensity may be associated with lower reliability than glintswith intensity within a certain middle intensity range (the suitablerange may for example depend on the image sensor or camera capturing theimages).

According to some embodiments, the determination of the first confidencevalue may be based at least in part on an evaluation of a similaritybetween an expected (or a predetermined) glint shape and a shape of aglint detected at the left eye.

The determination of the first confidence value may for example includeassociating a first level of glint shape similarity between the shapeswith a lower reliability than a reliability associated with a secondlevel of glint shape similarity between the shapes. The first level ofthe glint shape similarity may be lower than the second level of theglint shape similarity. In other words, gaze directions obtained using aglint which a shape similar to an expected glint shape may be assigned(or attributed) a higher reliability than reliabilities assigned to (orattributed to) gaze directions obtained using glints with shapes lesssimilar to the expected glint shape. In other words, the reliability maydecrease when the shape of the glint deviates more from the expectedglint shape. It will be appreciated that other factors may also affectthe reliability of the determined gaze directions, and the overallreliability may therefore increase even if the shape of the glintdeviates more from the expected glint shape.

A glint is typically caused by reflection of light from an illuminator.The shape of the glint depends on the shape of the illuminator and thegeometry of the surface at which the light is reflected. Therefore, theshape of the glint may be predicted. A deviation from the expected shapemay indicate that the glint is located at an area of the eye withunexpected optical properties (which may be detrimental to thereliability of the determined gaze direction) or that the glintoriginates from a different light source than expected. An unexpectedglint shape may therefore indicate lower reliability of the determinedgaze direction.

According to some embodiments, the determination of the secondconfidence value may be based at least in part on an evaluation of asimilarity between an expected glint shape and a shape of a glintdetected at the right eye.

The determination of the second confidence value may for example includeassociating a first level of glint shape similarity between the shapeswith a lower reliability than a reliability associated with a secondlevel of glint shape similarity between the shapes. The first level ofthe glint shape similarity may be lower than the second level of theglint shape similarity. In other words, gaze directions obtained using aglint which a shape similar to an expected glint shape may be assigned(or attributed) a higher reliability than reliabilities assigned to (orattributed to) gaze directions obtained using glints with shapes lesssimilar to the expected glint shape. In other words, the reliability maydecrease when the shape of the glint deviates more from the expectedglint shape. It will be appreciated that other factors may also affectthe reliability of the determined gaze directions, and the reliabilitymay therefore increase even if the shape of the glint deviates more fromthe expected glint shape.

According to some embodiments, the determination of the first confidencevalue may be based at least in part on a comparison between a number ofglints detected at the left eye and a predetermined (or expected) numberof glints.

The determination of the first confidence value may for example includeassociating a situation where the detected number of glints coincideswith the predetermined (or expected) number of glints with a higherreliability than a reliability associated with a situation where thedetected number of glints deviates from the predetermined (or expected)number of glints. In other words, gaze directions determined usingglints from an image with the predetermined number of glints may beassigned (or attributed) a higher reliability than reliabilitiesassigned to (or attributed to) gaze directions determined using glintsfrom an image with a different number of glints. In other words, thereliability may be decreased when the number of glints deviates from thepredetermined number of glints. It will be appreciated that otherfactors may also affect the reliability of the determined gazedirection, and that the overall reliability could therefore increaseeven if the detected number of glints deviates from the predeterminednumber of glints.

The reliability of the determined gaze directions may be increased byemploying glints from multiple illuminators. If some of the expectedglints are missing in the images, this may affect the reliability of thegaze tracking. If an unexpectedly high number of glints are present inthe images, it may be more difficult to identify the correct glints touse for the gaze tracking, and the incorrect glints may occlude imagedata of relevance for gaze estimation. An unexpectedly high number ofglints may for example be caused by other light sources in theenvironment and/or that a glint is about to fall off the cornea.

According to some embodiments, the determination of the secondconfidence value may be based at least in part on a comparison between anumber of glints detected at the right eye and a predetermined (orexpected) number of glints.

The determination of the second confidence value may for example includeassociating a situation where the detected number of glints at the righteye coincides with the predetermined (or expected) number of glints witha higher reliability than a reliability associated with a situationwhere the detected number of glints at the right eye deviates from thepredetermined (or expected) number of glints. In other words, gazedirections determined using glints from an image with the predeterminednumber of glints may be assigned (or attributed) a higher reliabilitythan reliabilities assigned to (or attributed to) gaze directionsdetermined using glints from an image with a different number of glints.In other words, the reliability may be decreased when the number ofglints deviates from the predetermined number of glints. It will beappreciated that other factors may also affect the reliability of thedetermined gaze direction, and that the overall reliability couldtherefore increase even if the detected number of glints deviates fromthe predetermined number of glints.

According to some embodiments, the determination of the first confidencevalue may be based at least in part on a parameter indicating presenceof a reflection from glasses in the one or more obtained images of theleft eye, and/or a parameter indicating whether a certain region of theleft eye is at least partially obscured in the one or more obtainedimages of the left eye. Similarly, the determination of the secondconfidence value may be based at least in part on a parameter indicatingpresence of a reflection from glasses in the one or more obtained imagesof the right eye, and/or a parameter indicating whether a certain regionof the right eye is at least partially obscured in the one or moreobtained images of the right eye.

Glints and/or pupils to be used in gaze tracking may for example bedrowned in reflections from glasses, whereby the reliability of thedetermined gaze direction may be affected. Computing the firstconfidence value may for example comprise associating a gaze directiondetermined during presence of a reflection from glasses in the left eyewith a lower reliability than a reliability associated with gazetracking data obtained without presence of such a reflection in the lefteye. Analogously, computing the second confidence value may for examplecomprise associating a gaze direction determined during presence of areflection from glasses in the right eye with a lower reliability than areliability associated with gaze tracking data obtained without presenceof such a reflection in the right eye.

Objects (such as eye lids or eye lashes) obscuring regions of an eye mayobscure features employed in gaze tracking, such as glints and/or thepupil, whereby the reliability of the determined gaze direction may bereduced. Gaze directions determined when certain regions of an eye areobscured may therefore be associated with (or assigned) lowerreliabilities than gaze directions obtained when such regions of the eyeare obscured.

According to some embodiments, the determination of the first confidencevalue and the second confidence value may be based at least in part on aparameter, for the left eye and the right eye respectively, indicating anumber of pupil edge pixels in the one or more obtained images of theeye. The pupil edge pixels may be image pixels located along (or locatedat) those one or more portions of the edge of the pupil which arevisible in the one or more obtained images of the eye.

The edge of the pupil may be employed to estimate the position of thecenter of the pupil, the edge of the pupil and the pupil size. This datamay be used for estimating a gaze direction and/or gaze point (or pointof regard) for the eye. If the number of image pixels located along (orat) the edge of the pupil is low, the reliability of the pupil position(and thereby the gaze direction determined based on the pupil position)may be reduced.

The determination of the first confidence value and the secondconfidence value may for example include associating a first number ofpupil edge pixels with a lower reliability than a reliability associatedwith a second number of pupil edge pixels. The first number of pupiledge pixels may for example be lower than the second of pupil edgepixels. In other words, a gaze direction determined based on images witha low number of pupil edge pixels may be assigned (or attributed) alower reliability than a gaze direction determined based on images witha high number of pupil edge pixels. In other words, the reliability mayincrease if the number of pupil edge pixels increases. It will beappreciated that other factors may also affect the reliability of thedetermined gaze direction, so that the overall reliability coulddecrease even if the number of pupil edge pixels increases.

According to some embodiments, the circuitry may be configured, for theleft eye and the right eye respectively, to determine, based on one ormore images of the eye, a first glint position at the eye caused byreflection of light from a first illuminator, to predict, based on thefirst glint position and a spatial relationship between the firstilluminator and a second illuminator, a second glint position at the eyecaused by reflection of light from the second illuminator, and todetermine, based on one or more images of the eye, a second glintposition at the eye caused by reflection of light from the secondilluminator.

The one or more or more images from which the first glint position isdetermined may for example be captured by a camera when the eye isilluminated by the first illuminator. The one or more or more imagesfrom which the second glint position is determined may for example becaptured when the eye is illuminated by the second illuminator.

The position of the second glint at the eye may for example be predictedbased on the position of the first glint at the eye and based onknowledge of a position and/or orientation of the second illuminatorrelative to the first illuminator, and optionally also based onknowledge of a position and/or orientation of the second illuminatorrelative to a camera arranged to capture the second image, a geometricmodel of the eye and/or an estimated position of the eye.

It will be appreciated that the fact that the position of the secondglint is “predicted” based on the first glint position does notnecessarily imply that the second glint appears after the first glint.The second glint may for example appear in the same image as the firstglint. The second glint may for example appear in an image before orafter the image in which the first glint appears.

According to some embodiments, the determination of the first confidencevalue and the second confidence value may be based at least in part on adifference (or distance) between the predicted second glint position andthe determined second glint position.

A large difference between the predicted position and the determinedposition of the second glint at an eye may indicate that the gazedirection determined for that eye may be less reliable than the gazedirection determined for the other eye.

Therefore, computing the first confidence value (and similarly thesecond confidence value, but for the right eye instead of for the lefteye) may for example comprise associating a first size of the differencebetween the predicted position and the determined position of the secondglint at the left eye with a lower reliability than a reliabilityassociated with a second size of the difference between the predictedposition and the determined position of the second glint at the lefteye. The first size of the distance may be larger than the second sizeof the difference. In other words, the gaze direction obtained usingimages where the determined and predicted positions of the second glintare close to each other may be assigned (or attributed) a higherreliability than reliabilities assigned to (or attributed to) gazedirections obtained using images where the predicted and determinedpositions of the second glint are further away from each other. In otherwords, the reliability may decrease when the distance between thepredicted and determined positions of the second glint increases. Itwill be appreciated that other factors may also affect the reliabilityof the determined gaze direction, and that the overall reliability couldpotentially increase even if the distance between the predicted anddetermined positions of the second glint increases.

According to some embodiments, the determination of the first confidencevalue and the second confidence value may be based at least in part on aparameter indicating whether the eye is dominant over the other eye.

The parameter indicating whether an eye is dominant over the other eyemay for example be computed based on how the respective eyes move and/orbased on respective deviations between the points of regard for therespective eyes and a reference point when the user is prompted to lookat the reference point.

According to some embodiments, the system may comprise at least oneilluminator for illuminating the eyes, and one or more cameras forcapturing images of the eyes.

At least one illuminator may for example be arranged coaxially with atleast one camera for capturing bright pupil images. At least oneilluminator may for example be arranged non-coaxially with at least onecamera for capturing dark pupil images.

According to some embodiments, the determination of the first and secondconfidence values may be based at least in part on parameters indicatinga magnitude of residual errors remaining after calibration of eye modelsemployed for computing gaze directions for the left and right eyes,respectively.

According to some embodiments, the circuitry may be configured toperform gaze tracking for the left eye for a sequence of images,estimate distances to the left eye (for example distances to the lefteye from a camera capturing the sequence of images) for the sequence ofimages, and estimate a noise level in the estimated distances to theleft eye. The circuitry may be configured to perform gaze tracking forthe right eye for a sequence of images, estimate distances to the righteye (for example distances to the right eye from a camera capturing thesequence of images) for the sequence of images, and estimate a noiselevel in the estimated distances to the right eye. The determination ofthe first and second confidence values may be based on parametersindicative of the noise levels in the estimated distances to the leftand right eyes, respectively.

A high noise level in the estimated distance to an eye may indicate thatthe determined gaze direction for that eye is unreliable. Therefore, thecomputation of the confidence value for an eye (such as the left and/orright eye) may for example comprise associating a first noise level witha lower reliability than a reliability associated with a second noiselevel. The first noise level may be higher than the second noise level.In other words, a gaze direction obtained when the noise level in theestimated distance is high may be assigned (or attributed) a lowerreliability than reliabilities assigned to (or attributed to) gazedirections obtained when the noise level in the estimated distance islow. In other words, the reliability may decrease when the noise levelin the estimated distance increases. It will be appreciated that otherfactors may affect the reliability of the determined gaze direction ofan eye, and that the overall reliability could therefore increase evenif the noise level in the estimated distance increases.

According to a second aspect, there is provided a method. The methodcomprises obtaining one or more images of a left eye of a user and oneor more images of right eye of the user. The method comprisesdetermining a gaze direction of the left eye of the user based on atleast one obtained image of the left eye, and determining a gazedirection of the right eye of the user based on at least one obtainedimage of the right eye. The method comprises, determining a firstconfidence value based on the one or more obtained images of the lefteye. The first confidence value represents an indication of thereliability of the determined gaze direction of the left eye. The methodcomprises determining a second confidence value based on the one or moreobtained images of the right eye. The second confidence value representsan indication of the reliability of the determined gaze direction of theright eye. The method comprises determining a final gaze direction basedat least in part on the first confidence value and second confidencevalue.

The method of the second aspect may for example be performed by thesystem of any of the embodiments of the first aspect, or by thecircuitry comprised in such systems.

Embodiments of the method according to the second aspect may for exampleinclude features corresponding to the features of any of the embodimentsof the system according to the first aspect.

According to a third aspect, there is provided one or morecomputer-readable storage media storing computer-executable instructionsthat, when executed by a computing system that implements eye/gaze dataprocessing, cause the computing system to perform a method. The methodmay for example be the method according to the second aspect.

Embodiments of the one or more computer-readable storage media accordingto the third aspect may for example include features corresponding tothe features of any of the embodiments of the system according to thefirst aspect.

The one or more computer-readable media may for example be one or morenon-transitory computer-readable media.

It is noted that embodiments of the invention relate to all possiblecombinations of features recited in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplifying embodiments will be described below with reference to theaccompanying drawings, in which:

FIG. 1 shows an eye tacking system, according to an embodiment;

FIG. 2 shows an example image of an eye;

FIG. 3 is a cross sectional view of a part of an eye;

FIGS. 4a-4g show examples of where glints from an illuminator may belocated at the left eye and right eye depending on where a user islooking;

FIG. 5 shows an example of a bright pupil image;

FIG. 6 shows an eye where a glint is located at the edge of the pupil;

FIGS. 7a and 7b show estimated gaze points before and after calibration;

FIG. 8 shows examples of low and high statistical variability;

FIG. 9 shows examples of low and high statistical bias;

FIGS. 10-12 are flow charts of methods, according to embodiments; and

FIG. 13 is a block diagram of a specialized computer system capable ofbeing used in at least some portion of the apparatuses or systems of thepresent invention, or implementing at least some portion of the methodsof the present invention.

All the figures are schematic and generally only show parts which arenecessary in order to elucidate the respective embodiments, whereasother parts may be omitted or merely suggested.

DETAILED DESCRIPTION OF THE INVENTION

The ensuing description provides exemplary embodiments only, and is notintended to limit the scope, applicability, or configuration of thedisclosure. Rather, the ensuing description of the exemplary embodimentswill provide those skilled in the art with an enabling description forimplementing one or more exemplary embodiments. It being understood thatvarious changes may be made in the function and arrangement of elementswithout departing from the spirit and scope of the invention as setforth herein.

For example, any detail discussed with regard to one embodiment may ormay not be present in all contemplated versions of that embodiment.Likewise, any detail discussed with regard to one embodiment may or maynot be present in all contemplated versions of other embodimentsdiscussed herein. Finally, the absence of discussion of any detail withregard to embodiment herein shall be an implicit recognition that suchdetail may or may not be present in any version of any embodimentdiscussed herein.

Specific details are given in the following description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, circuits,systems, networks, processes, and other elements in the invention may beshown as components in block diagram form in order not to obscure theembodiments in unnecessary detail. In other instances, well-knowncircuits, processes, algorithms, structures, and techniques may be shownwithout unnecessary detail in order to avoid obscuring the embodiments.

Also, it is noted that individual embodiments may be described as aprocess which is depicted as a flowchart, a flow diagram, a data flowdiagram, a structure diagram, or a block diagram. Although a flowchartmay describe the operations as a sequential process, many of theoperations can be performed in parallel or concurrently. In addition,the order of the operations may be re-arranged. A process may beterminated when its operations are completed, but could have additionalsteps not discussed or included in a figure. Furthermore, not alloperations in any particularly described process may occur in allembodiments. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination corresponds to a return of the functionto the calling function or the main function.

The term “machine-readable medium” or the like includes, but is notlimited to transitory and non-transitory, portable or fixed storagedevices, optical storage devices, wireless channels and various othermediums capable of storing, containing or carrying instruction(s) and/ordata. A code segment or machine-executable instructions may represent aprocedure, a function, a subprogram, a program, a routine, a subroutine,a module, a software package, a class, or any combination ofinstructions, data structures, or program statements. A code segment maybe coupled to another code segment or a hardware circuit by passingand/or receiving information, data, arguments, parameters, or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

Furthermore, embodiments of the invention may be implemented, at leastin part, either manually or automatically. Manual or automaticimplementations may be executed, or at least assisted, through the useof machines, hardware, software, firmware, middleware, microcode,hardware description languages, or any combination thereof. Whenimplemented in software, firmware, middleware or microcode, the programcode or code segments to perform the necessary tasks may be stored in amachine readable medium. A processor(s) may perform the necessary tasks.

Eye Tracking

FIG. 1 shows an eye tacking system 100 (which may also be referred to asa gaze tracking system), according to an embodiment. The system 100comprises illuminators 111 and 112 for illuminating the eyes of a user,and a light sensor 113 for capturing images of the eyes of the user. Theilluminators 111 and 112 may for example be light emitting diodesemitting light in the infrared frequency band, or in the near infraredfrequency band. The light sensor 113 may for example be a camera, suchas a complementary metal oxide semiconductor (CMOS) camera or a chargedcoupled device (CCD) camera.

A first illuminator 111 is arranged coaxially with (or close to) thelight sensor 113 so that the light sensor 113 may capture bright pupilimages of the user's eyes. Due to the coaxial arrangement of the firstilluminator 111 and the light sensor 113, light reflected from theretina of an eye returns back out through the pupil towards the lightsensor 113, so that the pupil appears brighter than the iris surroundingit in images where the first illuminator 111 illuminates the eye. Asecond illuminator 112 is arranged non-coaxially with (or further awayfrom) the light sensor 113 for capturing dark pupil images. Due to thenon-coaxial arrangement of the second illuminator 112 and the lightsensor 113, light reflected from the retina of an eye does not reach thelight sensor 113 and the pupil appears darker than the iris surroundingit in images where the second illuminator 112 illuminates the eye. Theilluminators 111 and 112 may for example take turns to illuminate theeye, so that every second image is a bright pupil image, and everysecond image is a dark pupil image.

The eye tracking system 100 also comprises circuitry 120 (for exampleincluding one or more processors) for processing the images captured bythe light sensor 113. The circuitry 120 may for example be connected tothe light sensor 113 and the illuminators 111 and 112 via a wired or awireless connection. In another example, circuitry 120 in the form ofone or more processors may be provided in one or more stacked layersbelow the light sensitive surface of the light sensor 113.

FIG. 2 shows an example of an image of an eye 200, captured by the lightsensor 113. The circuitry 120 may for example employ image processing(such as digital image processing) for extracting features in the image.The circuitry 120 may for example employ pupil center cornea reflection(PCCR) eye tracking to determine where the eye 200 is looking. In PCCReye tracking, the processor 120 estimates the position of the center ofthe pupil 210 and the position of the center of a glint 220 at the eye200. The glint 220 is caused by reflection of light from one of theilluminators 111 and 112. The processor 120 calculates where the user isin space using the glint 220 and where the user's eye 200 is pointingusing the pupil 210. Since there is typically an offset between theoptical center of the eye 200 and the fovea, the processor 120 performscalibration of the fovea offset to be able to determine where the useris looking. The gaze directions obtained from the left eye and from theright eye may then be combined to form a combined estimated gazedirection (or viewing direction). As will be described below, manydifferent factors may affect how the gaze directions for the left andright eyes should be weighted relative to each other when forming thiscombination.

In the embodiment described with reference to FIG. 1, the illuminators111 and 112 are arranged in an eye tracking module 110 placed below adisplay watched by the user. This arrangement serves only as an example.It will be appreciated that more or less any number of illuminators andlight sensors may be employed for eye tracking, and that suchilluminators and light sensors may be distributed in many different waysrelative to displays watched by the user. It will be appreciated thatthe eye tracking scheme described in the present disclosure may forexample be employed for remote eye tracking (for example in a personalcomputer, a smart phone, or integrated in a vehicle) or for wearable eyetracking (such as in virtual reality glasses or augmented realityglasses).

Glint Position on Cornea

There is an industry trend towards larger screens. While there exist eyetrackers supporting screens up to 27″ (16:9), it would be desirable tosupport even larger screens. A problem with supporting large screens isthat when the user watches a point close to the edge of the screen,glints tend to fall off the cornea so that it becomes difficult todetermine the point of regard of the user (which is also referred to asthe gaze point).

However, the glint on the right eye often does not fall off the corneaat the same time as the glint on the left eye. This means that even ifgaze tracking data from one of the eyes is not very useful (or not veryreliable), the gaze tracking data from the other eye may still be ofgood quality. Today, a gaze point at the screen is typically formed as asimple average of a gaze point for the left eye and a gaze point for theright eye. Therefore, the user experience is limited by the worst eye,and not by the best eye. If the position of the glint at each eye istaken into account during weighting of gaze tracking data from the leftand right eye, it is possible to provide a user experience more in linewith the gaze tracking data of the best eye. The gaze tracking data ofthe eyes may for example include gaze directions for the respective eyesand/or gaze points of the respective eyes.

FIG. 3 shows a cross section of different parts of an eye 300. Thecornea 310 has a central region 311 which in general is close tospherical, and an outer region 312 which is non-spherical and thereforemore difficult to model. The central region 311 (which is also referredto herein as the spherical region of the cornea 310) extends from thecornea center 313 to the edge 314 of the spherical region 314. Thecornea center 313 is a position at the spherical region 311 of thecornea 310 where a virtual line 340 extending from the eyeball center330 through the pupil center 350 intersects the spherical region 311 ofthe cornea 310. The outer region 312 of the cornea 310 (which is alsoreferred to herein as the non-spherical region of the cornea 310)extends from the edge 314 of the spherical region 311 of the cornea 310to the edge 315 of the cornea 310. FIG. 3 also shows the sclera 320 ofthe eye 300

In order to determine the gaze point of the eye 300 you need to find theuser position in space, and this is done by using the position of theglint and the geometry of the eye. The glint is also used fordetermining the rotational angle of the eye. Even if the glint has notfallen off the cornea 310 completely, it might be located in thenon-spherical region 312 of the cornea 310 where the shape varies muchmore between individuals. The shape of the non-spherical region 312 ofthe cornea 310 may even vary between the left and right eye of a person.

FIGS. 4a-4g show examples of where glints from an illuminator may belocated at the left eye and right eye, depending on which part of thescreen 420 a user 410 is looking at.

In these figures, the screen 420 is shown at the top right corner, andthe user's actual gaze point (or actual point of regard) is shown by thecircle 421. The dots 422 on the screen 420 represent reference points atwhich the user may look, for example during calibration. Each of theFIGS. 4a-4g shows the position of the glint 411 at the left eye 412 andthe position of the glint 413 at the right eye 414.

In FIG. 4a , the user 410 is looking straight into the camera which islocated below the screen 420. Both glints 411 and 413 are on thespherical region 311 of the cornea. The glints at both eyes thereforeseem suitable for gaze tracking.

In FIG. 4b , the user's gaze point 421 has moved upwards and to theright on the screen 420. Both the left eye glint 411 and the right eyeglint 413 are still in the spherical region 311 of the cornea. Theglints at both eyes therefore seem well suited for gaze tracking.

In FIG. 4c , the user's gaze point 421 has moved further upwards and tothe right on the screen 420. The left eye glint 411 is now in thenon-spherical region 312 of the cornea while the right eye glint 413 isstill in the spherical region 311 of the cornea. Therefore, the glint413 at the right eye seems well suited for gaze tracking while the glint411 at the left eye would make gaze tracking less reliable.

In FIG. 4d , the user's gaze point 421 has moved further upwards and tothe right on the screen 420. The left eye glint 411 is in thenon-spherical region 312 of the cornea while the right eye glint 413 isin the spherical region 311 of the cornea. Therefore, the glint 413 atthe right eye seems well suited for gaze tracking while the glint 411 atthe left eye would make gaze tracking less reliable.

In FIG. 4e , the user's gaze point 421 has moved further upwards on thescreen 420. The left eye glint 411 is close to the edge of the cornea310 while the right eye glint 413 is in the non-spherical region 312 ofthe cornea. Since the left eye glint 411 is about to fall off the cornea310, the geometry of the eye causes several glints to appear at the lefteye 412 instead of only one. This makes gaze tracking based on the lefteye glint 411 unreliable, while the right eye glint 413 may be somewhatbetter suited for gaze tracking.

In FIG. 4f , the user's gaze point 421 has moved to the upper rightcorner the screen 420. The left eye glint 411 is now on the sclera 320(which can be seen via the non-circular shape of the glint as well asthe existence of an additional glint further out on the sclera) whilethe right eye glint 413 is in the non-spherical region 312 of thecornea.

In FIG. 4g , the user's gaze point has moved even further upwards and tothe right. The left eye glint 411 is on the sclera 320 while the righteye glint 413 is now on the edge of the cornea 310. Since the right eyeglint 413 is about to fall off the cornea 310, the geometry of the eyecauses several glints to appear at the right eye 414 instead of onlyone.

As described above, the position of the glints on the cornea may differbetween the eyes. This is a factor which may affect the reliability ofgaze tracking data (such as gaze directions) determined based on theglints on the left and right eyes, respectively, and is listed as factorA in Table 1 below. This factor A may for example be expressed as thedistance between the glint and the cornea center 313. Since thespherical region 311 of the cornea 310 lies at the center of the cornea310 close to the pupil, the position of the glint at the cornea 310 mayalso be expressed as the position of the glint relative to the pupil ofthe eye (or relative to the pupil center 350). The factor A may forexample be expressed as the distance between the glint at the eye andthe center 350 of the pupil of the eye.

Pupil/Iris Contrast

The contrast between the pupil and the iris depends both on thereflection of the iris and the bright pupil effect of the user. Thebright pupil effect may differ significantly depending on age,ethnicity, pupil size, and eye tracker geometry, but may also differ forseveral other reasons. This means that the pupil/iris contrast may forexample be bad at one eye all the time, or the pupil/iris contrast maybe ok at both eyes but may suddenly disappear for one eye. The brightpupil effect may for example disappear for certain angles. When there islow (or no contrast) between the pupil and the iris, it may be difficultto detect the pupil, whereby the gaze direction cannot be determinedcorrectly. If this happens for one eye, then the gaze tracking data(such as the gaze direction) for that eye should be given a lower weight(or confidence) than gaze point data for the other eye. In other words,pupil/iris contrast is a factor affecting the reliability of gazetracking data (such as determined gaze directions) for the left andright eye, and is listed as factor B in Table 1 below.

FIG. 5 shows an example where the pupil 511 of the right eye 510 isclearly visible as a bright spot while it is more or less impossible todetermine the pupil center in the left eye 520 due to the low pupil/iriscontrast.

Glint Intensity

If either the pupil or the glint is determined incorrectly, theestimated gaze point on the screen may be off. Even small errors mayresult in a bad eye tracking experience. For the position of the glintto be determined accurately it may be important that the glint is notover saturated in the images, and that the glint does not have too lowintensity. Saturation may cause the glint to appear as a large whiteblob in the image. The center of the glint may be difficult to estimatesince saturation may cause an entire region to have the same color asthe center of the glint. Possible reasons for low/high intensity glintsmay be eye surgery or variations in thickness of the tear film on theeye. Glint intensity is a factor affecting the reliability of gazetracking data (such as determined gaze directions) for the left andright eye, and is listed as factor C in Table 1 below.

Glint on Pupil Edge

FIG. 6 shows an eye 600 where the glint 610 from an illuminator islocated at the edge 620 of the pupil 630. The glint 610 overlaps thepupil edge 620 and therefore causes a smaller part of the edge 620 to bevisible in the image, making estimation of the pupil center moreunreliable. The size of the glint 610 relative to the pupil 630 may bedifferent in different situations. If the glint 610 is large relative tothe pupil 630 it may cover a significant portion of the edge 620 of thepupil 630.

Moreover, in bright pupil images, calculation of the center of the glint610 may be affected by the bright pupil 630 if the glint 610 is locatedat the edge 620 of the pupil 630. Calculation of the center of the glint610 may for example be performed similarly to calculating a center ofmass, but with light intensity instead of mass. The calculated glintposition can therefore be “pulled” into the pupil 630, which may resultin an incorrectly determined user position and/or gaze angle.

In view of the above, if one of the eyes has a glint 610 on the pupiledge 620 (or has a glint 610 overlapping the pupil edge 620), gazetracking data from that eye should be weighted lower than gaze trackingdata from the other eye, especially if the glint 610 covers a largeportion of the pupil edge 620. Glint on pupil edge is a factor affectingthe reliability of gaze tracking data (such as determined gazedirections) for the left and right eye, and is listed as factor D inTable 1 below.

Glasses Reflections

Wearing glasses may result in a big reflection in images of an eye. Ifsuch a reflection covers the eye in a bad way (for example coveringcertain regions such as the pupil), it may be difficult or evenimpossible to find the pupil center and/or the glint in the image. Ifthis happens for one eye, it may be desirable to ignore gaze trackingdata from that eye completely. On the other hand, a glasses reflectioncould in some cases cover only parts of the eye, so that the pupil andglint may still be detected, but with lower accuracy than normal. Insuch cases, it may be desirable to use the gaze tracking data from thateye, but with lower weight than for the other eye. Eye trackers may bedesigned so that the probability of both eyes having reflections fromglasses at the same time is very low (for example by placing theilluminators and cameras at suitable locations). Hence, the impact ofreflections from glasses may be reduced by weighting the gaze trackingdata (such as the gaze directions) from the left and right eyesappropriately. Presence of a glasses reflection is a factor affectingthe reliability of gaze tracking data (such as the determined gazedirections) for the left and right eyes, and is listed as factor E inTable 1 below.

Obscuring Objects

Objects such as eye lashes or eye lids may in some cases cover crucialparts of any eye, making it difficult to determine the gaze point (orgaze direction) for that eye. In such situations, it may be desirable togive gaze tracking data (such as the gaze direction) from that eye lowerweight (or a lower confidence) than gaze tracking data from the othereye. In some cases where critical areas of the eye are obscured, it mayeven be desirable to ignore the gaze tracking data (or the gazedirection) from that eye completely, in favor of the gaze tracking data(or gaze direction) from the unobscured eye. Presence of obscuringobjects is a factor affecting the reliability of gaze tracking data(such as determined gaze directions) for the left and right eyes, and islisted as factor F in Table 1 below.

Machine Learning Confidence

A black-box machine learning algorithm may be employed to generate aconfidence value for each eye. Suitably designed machine learningalgorithms could potentially yield appropriate confidence values ifsufficient amounts of training data are made available to the algorithm.Such a confidence value may be employed as a factor indicative of thereliability of gaze tracking data (such as determined gaze directions)for the left and right eye, and is listed as factor G in Table 1 below.The machine learning algorithm(s) may for example be trained based oncomparisons between determined gaze tracking data and correspondingreference points/positions where the user is actually looking.

Number of Pupil Edge Pixels

The pupil center may be computed using those parts of the pupil edgewhich are visible in the image. The images are typically digital imagesfor which pixels located along (or covering) the pupil edge are employedto estimate the position of the pupil center. Determining the pupilcenter gets easier the more pupil edge pixels you have available in theimage. Many factors may affect the number of pupil edge pixels, such aspupil size, occlusion etc. Basically, it is desirable to use imageswithout anything occluding/obscuring the pupil, and it is desirable tohave as many pupil edge pixels as possible. If one eye has less pupiledge pixels, it may therefore be desirable to give gaze tracking data(such as a determined gaze direction) for that eye lower weight (or alower confidence) than gaze tracking data from the other eye. The numberof pupil edge pixels is a factor affecting the reliability of gazetracking data (such as determined gaze directions) for the left andright eye respectively, and is listed as factor H in Table 1 below.

Spatial Distribution of Pupil Edge Pixels

The spatial distribution of pupil edge pixels may also affect how wellthe pupil center position, the pupil size, and/or the pupil shape may beestimated. For example, x pupil edge pixels located only at the lefthalf of the pupil may result in less reliable gaze tracking data (suchas a less reliable gaze direction) than if x/2 pupil edge pixels wereavailable at the left half of the pupil and x/2 pupil edge pixels wereavailable at the right half of the pupil. In other words, the degree ofspatial distribution of the pupil edge pixels in the one or more imagesobtained of the respective eyes is therefore a factor affecting thereliability of gaze tracking data (such as determined gaze directions)for the left and right eyes respectively, and is listed as factor Q inTable 1 below. The factor Q may also be expressed as the degree ofspatial distribution of the available pupil edge pixels along the pupiledge of the eye, or as the angular distribution of the pupil edge pixelsabout the center of the pupil.

Calibration Quality (Residuals)

In order to provide a good eye tracking experience, a calibration isneeded. FIGS. 7a-7b illustrate calibration of the eye tracker. Duringthe calibration procedure the user looks at test points on the screenand algorithms optimize the eye model in order to minimize the error foreach of these points. As illustrated in FIG. 7a , the user looks at anumber of test points 701. Gaze points 702 are estimated by the eyetracker for each of the tests points 701. In the present example, tenconsecutive gaze points 702 are generated for each test point 701. Theeye model is then calibrated to minimize the deviations between the testpoints 701 and the associated estimated gaze points 702 (for example tominimize the sum of squares of the deviations). The result aftercalibration is shown in FIG. 7b , where the new gaze points 703 arecloser to the test points 701. The remaining deviations (or errors) maybe referred to as residual errors, or simply residuals. The size of theresiduals may be regarded as a measure of the calibration quality. Thecalibration is performed for each eye, and the residuals are typicallydifferent for each eye. If the gaze data (such as the determined gazedirection) for one eye is associated with large residuals, that mayindicate that the gaze tracking data for an eye is less reliable, and itmay be desirable to give the gaze tracking data for that eye lowerweight (or lower confidence) than for the other eye. In other words, thesize of the residuals (or the calibration quality) is a factor affectingthe reliability of the gaze tracking data (such as the determined gazedirections) for the left and right eye respectively, and is listed asfactor I in Table 1 below.

In Table 1, the factor I is expressed as a parameter indicating amagnitude of residual errors remaining after calibration of an eye modelemployed for computing gaze tracking data for the eye. The residualerrors may be of different types. In the example described above withreference to FIGS. 7a and 7b , the deviations between the test points701 and the estimated gaze points 702 are minimized, and the residualerror is a measure (for example the sum of squares) of the remainingdeviations. Other possible quantities to use in the calibration includedeviations between the actual eye position and estimated eye positions,or deviations between the actual pupil size and estimated pupil sizes.If the calibration is performed to minimize such other quantities, thenthe resulting residual error may for example be formed as the sum ofsquares of what remains of these quantities (or deviations) after thecalibration.

As described above, the magnitude of the residual errors remaining aftercalibration for an eye (such as the left and/or right eye) may beindicative of the reliability of the gaze tracking data obtained duringgaze tracking of that eye after the calibration. For example, computinga confidence value for an eye (such as the left eye and/or the righteye) may comprise associating a first magnitude of residual errors witha lower reliability than a reliability associated with a secondmagnitude of residual errors. The first magnitude of residual errors maybe higher than the second magnitude of residual errors.

Glint Association Confidence

An eye tracker employing multiple illuminators, such as the eye trackingsystem 100 described above with reference to FIG. 1, may employ a glintfrom a first illuminator 111 to predict where a glint from a secondilluminator 112 should appear. The eye tracker 100 may for exampleemploy the first illuminator 111 to illuminate an eye (or both eyes)during a first image frame, and employ the second illuminator 112 toilluminate the eye (or both eyes) during a second image frame which isthe frame after the first image frame. The image captured by the lightsensor 113 at the first image frame will include a glint at the eyecaused by reflection of light from the first illuminator 111. Since sucha short time has passed between the first image frame and the secondimage frame, the user will typically not move that much between theseframes. Since the position of the second illuminator 112 relative to thefirst illuminator 111 and the light sensor 113 is known, it is thereforepossible to predict where a glint caused by reflection of light from thesecond illuminator 112 should appear in the image captured at the nextimage frame, by using the position of the glint from the firstilluminator 111 in the first image frame. Such a prediction may also bebased on a geometric model of the eye and an estimated position of theeye relative to the illuminators 111 and 112 and the light sensor 113.

Several glints may sometimes appear in an image, and there is a riskthat the wrong glint is detected in an image, leading to errors in theestimated gaze point. The ability to predict where a glint should belocated allows the eye tracker to check whether the detected glint (orthe determined glint position) appears to be the correct one. In otherwords, the closer the detected glint is to the predicted position, thelarger the chance of it being the correct glint. The difference betweenthe predicted position and the position actually detected (or actuallydetermined or estimated) in the image is a factor indicative of thereliability of the gaze tracking data (such as determined gazedirections) for the left and right eye respectively, and is listed asfactor J in Table 1 below.

It will be appreciated that even if the correct glint has been detected,the detected position may still deviate from the predicted position. Ifsuch a deviation is large, it may indicate that something else is wrongand/or unreliable with the gaze tracking data for that eye. Gazetracking data for the eye with a large deviation between predicted glintposition and detected glint position may therefore be provided with alower weight (of confidence) than the gaze tracking data of the othereye.

It will also be appreciated that the two illuminators 111 and 112 couldfor example be employed to illuminate an eye in the same image frame.The position of the glint from one of the illuminators in an image maythen be employed to predict the position of the glint from the otherilluminator in the same image. In other words, the “prediction” need notnecessarily be performed from one image to a subsequent image. In fact,the prediction could also be performed backwards, from one image to aprevious image.

Ocular Dominance

If the user has a dominant eye, gaze tracking data from that eye may begiven higher weight (or higher confidence) than gaze tracking from theother eye. The dominant eye may for example be detected by analyzing howthe left and right eyes move and/or by analyzing the actual gaze pointsof the left and right eyes when the user looks at test points. Oculardominance is a factor indicative of the reliability of the gaze trackingdata (such as determined gaze directions) for the left and right eyerespectively, and is listed as factor K in Table 1 below.

Even if the dominant eye may be detected by analyzing images captured bythe eye tacker, it could also be known a priori. For example, the usercould enter this information manually.

Gaze Precision (Aggregation Level)

Precision is a measure of statistical variability, or how close togethergaze points are aggregated/clustered. An example of bad precision (orhigh statistical variability, or large random errors) is shown to theleft in FIG. 8 where the estimated gaze points 801 are distributed overa relatively large region rather than being aggregated/clusteredtogether in a small region. An example of better precision is shown tothe right in FIG. 8 where the estimated gaze points 803 are located (oraggregated/clustered) closer together. The position of the estimatedgaze points 801 and 803 relative to the true gaze point of the user(indicated in FIG. 8 by 802 and 804) is irrelevant for precision.

During gaze tracking, the gaze precision for each eye (that is, theprecision of the gaze direction determined for the left eye and theright eye, respectively) may be calculated continuously to monitor ifthere is anything in the images that makes it difficult for thealgorithms to find the pupil and the glint correctly. If the precisionfor one eye becomes worse than for the other eye, there is probablysomething in the image that the algorithms cannot handle. It maytherefore be desirable to apply a lower weight (or confidence) for thegaze tracking data for the eye with the bad precision than for the gazetracking data for the other eye. Gaze precision is a factor indicativeof the reliability of the gaze tracking data for the left and right eyerespectively, and is listed as factor L in Table 1 below. The factor Lmay also be expressed as an aggregation level of determined gazepositions.

When the user is looking at a fixed point, the determined gaze directionfor both eyes should be constant. During saccades (when the user movestheir gaze to a new point), the determined gaze directions for both eyesshould change. Changes in the determined gaze directions due to saccadescould potentially be mistaken for bad gaze precision for both eyes.However, changes in the determined gaze direction for a first eye whilethe determined gaze direction for the second eye is constant may be anindication that the precision for the first eye is bad.

Gaze Accuracy

Accuracy is a measure of statistical bias (or systematic errors). Anexample of bad accuracy (or large statistical bias) is shown to the leftin FIG. 9 where the estimated gaze points 901 are distributed in aregion above the true gaze point 902 of the user (the average of theestimated gaze points 901 is located above the true gaze point 902). Anexample of better precision (or small statistical bias) is shown to theright in FIG. 9 where the estimated gaze points 903 are distributedabout the true gaze point 904 of the user so that the average of theestimated gaze points 903 is located close to the true gaze point 904.If the accuracy is worse for one eye, it may be desirable to apply alower weight (or confidence) to the gaze tracking data (or thedetermined gaze direction) for that eye, than to the gaze tracking data(or the determined gaze direction) of the other eye having higheraccuracy. Gaze accuracy is a factor indicative of the reliability of thegaze tracking data (such as determined gaze directions) for the left andright eye respectively, and is listed as factor M in Table 1 below. Thefactor M in table 1 may for example be expressed as a distance between apredetermined position (such as a reference position, or a true gazeposition) and an average of determined gaze positions for an eye.

The gaze accuracy may be determined continuously during gaze tracking,or during calibration.

Noise in User Distance

Glint positions may be employed for calculating the user distance.Errors in the glint position may cause the calculated user position inspace to be noisy, which will cause noise also in the estimated gazepoint (or gaze direction). In other words, a high noise level in thecalculated user distance is an indication that the gaze tracking data isunreliable. For each eye, the user position may be calculatedcontinuously. The noise level in the user distance calculated for eacheye may be monitored during the gaze tracking. If the noise level ishigh for the user distance calculated for one eye, it may be desirableto apply a lower weight (or confidence) to the gaze tracking data (ordetermined gaze direction) for that eye than to the gaze tracking data(or determined gaze direction) from the other eye. The noise level inthe calculated user distance is a factor indicative of the reliabilityof the gaze tracking data (such as determined gaze directions) for theleft and right eye respectively, and is listed as factor N in Table 1below.

Glint Shape

If a circularly shaped illuminator (or an illuminator which may beregarded as a point source) is employed to illuminate an eye, areflection at the spherical portion of the cornea will typically resultin a circularly shaped glint in the image. If the shape of the glint isnot circular, that may be an indication that the cornea has anon-spherical structure (such as a scar) or that the glint is no longeron the cornea at all. Both these scenarios may result in poor estimationof the user position, and a bad eye tracking experience.

Optics of the light sensor (or camera), and/or the shape of theilluminator may affect the glint shape in the images. Still, the glintshape in the images may be predicted to at least some extent (forexample if it is assumed that the glint is located at the sphericalportion of the cornea).

A similarity between the shapes of the detected glints and the expectedglint shape may be monitored for the left and right eyes. If anunexpected glint shape is detected for a first eye but not for a secondeye, then it may be desirable to apply a lower weight (or confidence) tothe gaze tracking data (or determined gaze direction) for the first eyethan for the gaze tracking data (or determined gaze direction) for thesecond eye. The glint shape is a factor indicative of the reliability ofthe gaze tracking data (such as determined gaze directions) for the leftand right eye respectively, and is listed as factor O in Table 1 below.

Number of Glints

When a large number of potential glints are found, the risk of choosingthe wrong one increases. This can for example happen when there areother light sources with infrared content in the environment, or whenthe glint is about to fall off the cornea. An unexpectedly high numberof glints may therefore be an indication that the gaze tracking datafrom that eye is unreliable.

In some eye trackers (for example in virtual reality eye trackers)multiple illuminators may be employed simultaneously, so that severalglints intentionally appear at the eye. The reliability of the gazetracking data may be reduced if some of the expected glints are missingin an image of the eye.

In view of the above, it may be desirable to provide a higher weight (orhigher confidence) to gaze tracking data (or a determined gazedirection) from an eye where the detected number of glints matches theexpected number of glints, than to gaze tracking data (or a determinedgaze direction) from an eye where the detected number of glints differsfrom the expected number of glints. The number of detected glints is afactor indicative of the reliability of the gaze tracking data (such asthe determined gaze directions) for the left and right eye respectively,and is listed as factor P in Table 1 below.

Example Embodiments

FIG. 10 is a flow chart of a method 1000 which may for example beperformed by the eye tracking system 100, described above with referenceto FIG. 1, or the circuitry 120 of the eye/gaze tracking system 100.

The method 1000 comprises obtaining 1001 one or more images of a lefteye of a user and one or more images of a right eye of the user. Theimages may be obtained by the image sensor 113.

The method 1000 comprises determining 1002 a gaze direction of the lefteye of the user based on at least one obtained image of the left eye,and determining 1003 a gaze direction of the right eye of the user basedon at least one obtained image of the right eye

The method 1000 comprises determining 1004 a first confidence valuebased on the one or more obtained images of the left eye. The firstconfidence value represents an indication of the reliability of thedetermined gaze point of the left eye.

The method 1000 comprises determining 1005 a second confidence valuebased on the one or more obtained images of the right eye. The secondconfidence value represents an indication of the reliability of thedetermined gaze point of the right eye.

The method 1000 comprises determining 1006 a final (or combined) gazedirection based on the determined gaze directions for the left and righteyes, and based on the first and second confidence values. Rather thandetermining a final gaze direction, the step 1006 could for exampledetermine a combined gaze point based on the determined gaze directionsfor the left and right eyes, and based on the first and secondconfidence values.

The first and second confidence values are determined (or computed) atsteps 1004 and 1005 based on one or more parameters representingrespective factors indicative of a reliability of the determined gazetracking data (or gaze directions) for the left and right eyes.

The parameter(s) employed to determine the confidence values at steps1004 and 1005 may include all of the parameters A-Q listed in Table 1below, or may include a subset of the parameters A-Q.

TABLE 1 Parameter Parameter representing a factor which is indicative ofa reliability label of gaze tracking data for the left and right eyesrespectively A a position of a glint at the eye relative to a cornea ofthe eye; a distance between a glint at the eye and a cornea center ofthe eye; a position of a glint at the eye relative to a pupil of theeye; or a distance between a glint at the eye and a center of a pupil ofthe eye B a contrast between the pupil and the iris of the eye in theone or more images of the eye C an intensity of a glint at the eye D aposition of a glint at the eye relative to an edge of a pupil of the eyeE a parameter indicating presence of a reflection from glasses in theone or more images of the eye F a parameter indicating whether a certainregion of the eye is at least partially obscured in the one or moreimages of the eye G a confidence value estimated by a machine learningalgorithm based on the one or more images of the eye H a parameterindicating a number of pupil edge pixels in the one or more images ofthe eye, wherein the pupil edge pixels are image pixels located alongthose one or more portions of the edge of the pupil which are visible inthe one or more images of the eye I a parameter indicating a magnitudeof residual errors remaining after calibration of an eye model employedfor computing gaze tracking data (such as gaze directions) for the eye Ja difference between a predicted position of a glint and a determined(or detected) position of the glint K a parameter indicating whether theeye is dominant over the other eye L an aggregation level of determinedgaze positions of the eye; or a parameter indicative of the precision orthe gaze tracking of the eye M a distance between a predeterminedposition at a display apparatus and an average of determined gazepositions/points for the eye; or a parameter indicative of the accuracyof the gaze tracking for the eye N a parameter indicative of a noiselevel in estimated distances to the eye for a sequence of images of theeye O a shape of a glint at the eye in the one or more images of the eyeP a number of glints detected in the one or more images of the eye Q aparameter indicating a degree of spatial distribution of pupil edgepixels in the one or more obtained images of the eye, wherein the pupiledge pixels are image pixels located along those one or more portions ofthe edge of the pupil which are visible in the one or more obtainedimages of the eye

The parameter(s) employed to determine the confidence values at steps1004 and 1005 may for example include the following parameters:

-   -   at least one of the parameters A-Q    -   at least two of the parameters A-Q    -   at least three of the parameters A-Q    -   at least four of the parameters A-Q    -   at least five of the parameters A-Q    -   A and at least one more of the parameters A-Q    -   B and at least one more of the parameters A-Q    -   H and at least one more of the parameters A-Q    -   A and B    -   A and H    -   A, B, and H

The first and second confidence values may be computed in differentways. The confidence value for an eye (such as the left and and/or theright eye) may for example be computed as a weighted average of thedifferent parameters computed for that eye. An explicit example will nowbe described. In the present example, the parameters A, B and H areemployed

Computation of the parameter A, relating to the position of the glintrelative to the cornea, is described with reference to FIG. 3. In thepresent example, the value of the parameter A is assigned for the leftand right eye independently of each other. The parameter A is assignedthe value 1 if the glint is in the spherical region 311 of the cornea310. If the glint is in the non-spherical region 312 of the cornea 310,the parameter A is assigned a value which decreases gradually from 1 to0 as the glint moves closer to the edge 315 of the cornea 310. When theglint is at the sclera 320, the parameter A is assigned the value 0.

Computation of the parameter B, relating to the pupil/iris contrast maybe performed as follows. In the present example, the value of theparameter B is assigned for the left and right eye independently of eachother. When the pupil intensity is not more than 1.3 times the irisintensity in a bright pupil image, the value assigned for the parameterB is 0. The value assigned to the parameter B increases linearly from 0to 1 as the pupil intensity increases to 2 times the iris intensity in abright pupil image. The value assigned to the parameter B remains at 1for higher contrast values. When the iris intensity is not more than 1.3times the pupil intensity in a dark pupil image, the value assigned forthe parameter B is 0. The value assigned to the parameter B increaseslinearly from 0 to 1 as the iris intensity grows to 2 times the pupilintensity in a dark pupil image. The value assigned to the parameter Bremains at 1 for higher contrast values.

Computation of the parameter H, relating to the number of pupil edgepixels may be performed as follows. In the present example, the value ofthe parameter H is assigned for the left and right eye relative to eachother (that is, the values of the parameter H assigned for the left andright eyes are not independent of each other). The value assigned forthe left eye may be formed as the number of pupil edge pixels for theleft eye divided by the number of pupil edge pixels for the right eye:

$H_{{left}\mspace{14mu}{eye}} = \frac{\#\mspace{11mu}{pupilEdgePixels}_{{left}\mspace{14mu}{eye}}}{\#\mspace{11mu}{pupilEdgePixels}_{{right}\mspace{14mu}{eye}}}$

Similarly, the value assigned for the right eye may be formed as thenumber of pupil edge pixels for the right eye divided by the number ofpupil edge pixels for the left eye:

$H_{{right}\mspace{14mu}{eye}} = \frac{\#\mspace{11mu}{pupilEdgePixels}_{{right}\mspace{14mu}{eye}}}{\#\mspace{11mu}{pupilEdgePixels}_{{left}\mspace{14mu}{eye}}}$

The confidence value for the left eye may then be computed as:

$C_{{left}\mspace{14mu}{eye}} = \frac{A_{{left}\mspace{14mu}{eye}} + B_{{left}\mspace{14mu}{eye}} + H_{{left}\mspace{14mu}{eye}}}{\begin{matrix}{A_{{left}\mspace{14mu}{eye}} + B_{{left}\mspace{14mu}{eye}} + H_{{left}\mspace{14mu}{eye}} +} \\{A_{{right}\mspace{14mu}{eye}} + B_{{right}\mspace{14mu}{eye}} + H_{{right}\mspace{14mu}{eye}}}\end{matrix}}$

Similarly, the confidence for the right eye may be computed as

$C_{{right}\mspace{14mu}{eye}} = \frac{A_{{right}\mspace{14mu}{eye}} + B_{{right}\mspace{14mu}{eye}} + H_{{right}\mspace{14mu}{eye}}}{\begin{matrix}{A_{{left}\mspace{14mu}{eye}} + B_{{left}\mspace{14mu}{eye}} + H_{{left}\mspace{14mu}{eye}} +} \\{A_{{right}\mspace{14mu}{eye}} + B_{{right}\mspace{14mu}{eye}} + H_{{right}\mspace{14mu}{eye}}}\end{matrix}}$

As can be seen in the equations above, the confidence values for theleft and right eye are normalized so that the sum of the confidencevalues is 1. This makes the confidence values suitable as weights if aweighted average of the gaze tracking data (such as gaze directions orgaze points) for the left and right eye is to be formed.

As illustrated by the above example, the values of the parameters A-Qmay for example be computed for the left and right eyes independently ofeach other (like the parameters A and B in the example above), or may becomputed for the left and right eyes relative to each other (like theparameter H in the example above).

It will be appreciated that the example given above is merely one ofmany different ways to compute confidence values using parameters fromTable 1 above.

The step of determining 1006 a final gaze direction, may for exampleinclude forming a weighted combination of the gaze direction for theleft eye and the gaze direction for the right eye. When forming theweighted combination, the gaze directions for the respective eyes areweighted by the respective associated confidence values. In other words,the right eye gaze direction is weighted by the right eye confidencevalue and the left eye gaze direction is weighted by the left eyeconfidence value. In this way, a combined (or final) gaze direction isobtained as a weighted combination of the estimated gaze directions forthe left and right eyes.

Since this weighed combination takes into account one or more of thefactors A-Q for the left and right eyes, the negative impact of errorspresent only in the gaze tracking data (or determined gaze direction) ofone eye may be more efficiently mitigated than if the left and right eyegaze tracking data (or gaze directions) were combined without takingsuch factors into account.

The confidence values for the left eye and the right eye determined atthe steps 1004 and 1005 may for example be output by the eye trackingsystem 100 together with the gaze tracking data (or gaze directions) forthe left and right eyes. As described below with reference to FIG. 12, asystem receiving such signals may then itself determine (or compute) thecombined gaze direction.

Rather than determining 1004 the first confidence value and determining1005 the second confidence value, both based on one or more of theparameters A-Q, such parameters A-Q may instead be output by the eyetracking system 100 together with gaze tracking data (or gazedirections) for the left and right eyes. As described below withreference to FIG. 11, a system receiving such signals may then itselfdetermine (or compute) confidence values for the left and right eyes,and then determine (or compute) the combined gaze direction.

FIG. 11 is a flow chart of a method 1100 which may be performed by asystem receiving data from the gaze tracking system 100 described abovewith reference to FIG. 1. The method 1100 may for example be performedby an operating system, or by a one or more processors.

The method 1100 comprises receiving 1101 gaze tracking data indicating agaze direction and/or gaze point of a left eye, and receiving 1102 gazetracking data indicating a gaze direction and/or gaze point of a righteye.

The method 1100 comprises receiving 1103 a plurality of parametersrepresenting different factors indicative of a reliability of the gazetracking data for the left eye, and receiving 1104 a plurality ofparameters representing different factors indicative of a reliability ofthe gaze tracking data for the right eye. The plurality of parametersreceived at steps 1103 and 1104 may be any of the parameters A-Q inTable 1 above.

The method 1100 comprises computing 1105 (or determining) a combinedreliability parameter (or first confidence value) for the gaze trackingdata of the left eye based on at least one of the received parametersassociated with the left eye, and computing 1106 a combined reliabilityparameter (or second confidence value) for the gaze tracking data of theright eye based on at least one of the received parameters associatedwith the right eye. The steps 1105 and 1106 of the method 1100 may forexample be identical to the steps 1004 and 1005 of the method 1000,described above with reference to FIG. 10.

The system performing the method 1100 may for example be able to selectwhich of the parameters received at the steps 1103 and 1104 to employwhen computing the confidence values for the left and right eyes at thesteps 1105 and 1106. For example, any eye tracking system 100 mayprovide all the parameters A-Q, but the system performing the method1100 may select only to use a subset of these parameters, such as theparameters A, B and H, when forming the confidence values for the leftand right eyes.

Optionally, the method 1100 may comprise forming 1107 (or determining) aweighted combination of the gaze tracking data (or the determined gazedirections) for the left eye and the gaze tracking data (or thedetermined gaze directions) for the right eye. When the weightedcombination is formed, the gaze tracking data for the respective eyesare weighted based on the respective associated confidence values. Thestep 1107 of the method 1100 may for example be identical to the step1006 of the method 1000 described above with reference to FIG. 10. Thestep 1107 may for example yield a combined gaze direction or a combinedgaze point.

FIG. 12 is a flow chart of a method 1200 which may be performed by asystem receiving data from the gaze tracking system 100 described abovewith reference to FIG. 1. The method 1200 may for example be performedby an operating system, or by a one or more processors.

Just like the method 1100 described above with reference to FIG. 11, themethod 1200 comprises receiving 1101 gaze tracking data indicating agaze direction and/or gaze point of a left eye, and receiving 1102 gazetracking data indicating a gaze direction and/or gaze point of a righteye.

Instead of receiving parameters from Table 1 above, the method 1200comprises receiving 1203 the confidence value (or the combinedreliability parameter) for the gaze tracking data of the left eye (forexample computed at the step 1004 of the method 1000), and receiving1204 the confidence value (or the combined reliability parameter) forthe gaze tracking data of the right eye (for example computed at thestep 1005 of the method 1000).

Just like the method 1100 described above with reference to FIG. 11, themethod 1200 comprises forming 1107 (or determining, or computing) aweighted combination of the gaze tracking data (or gaze direction) forthe left eye and the gaze tracking data (or the gaze direction) for theright eye.

Miscellaneous

The person skilled in the art realizes that the present invention is byno means limited to the preferred embodiments described above. On thecontrary, many modifications and variations are possible within thescope of the appended claims. For example, the person skilled in the artrealizes that the eye/gaze tracking methods described herein may beperformed by many other eye/gaze tracking systems than the exampleeye/gaze tracking system 100 shown in FIG. 1, for example using multipleilluminators and multiple cameras. It will be appreciated that any othercombinations of the parameters A-Q in Table 1 than those explicitlymentioned herein may also be employed for determining confidence values(or combined reliability parameters) for the left end right eyes.

Additionally, variations to the disclosed embodiments can be understoodand effected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. The division of tasks between functional unitsreferred to in the present disclosure does not necessarily correspond tothe division into physical units; to the contrary, one physicalcomponent may have multiple functionalities, and one task may be carriedout in a distributed fashion, by several physical components incooperation. A computer program may be stored/distributed on a suitablenon-transitory medium, such as an optical storage medium or asolid-state medium supplied together with or as part of other hardware,but may also be distributed in other forms, such as via the Internet orother wired or wireless telecommunication systems. The mere fact thatcertain measures/features are recited in mutually different dependentclaims does not indicate that a combination of these measures/featurescannot be used to advantage. Method steps need not necessarily beperformed in the order in which they appear in the claims or in theembodiments described herein, unless it is explicitly described that acertain order is required. Any reference signs in the claims should notbe construed as limiting the scope.

Specialized Computer System

FIG. 13 is a block diagram illustrating a specialized computer system1300 in which embodiments of the present invention may be implemented.This example illustrates specialized computer system 1300 such as may beused, in whole, in part, or with various modifications, to provide thefunctions of components described herein.

Specialized computer system 1300 is shown comprising hardware elementsthat may be electrically coupled via a bus 1390. The hardware elementsmay include one or more central processing units 1310, one or more inputdevices 1320 (e.g., a mouse, a keyboard, eye tracking device, etc.), andone or more output devices 1330 (e.g., a display device, a printer,etc.). Specialized computer system 1300 may also include one or morestorage device 1340. By way of example, storage device(s) 1340 may bedisk drives, optical storage devices, solid-state storage device such asa random access memory (“RAM”) and/or a read-only memory (“ROM”), whichcan be programmable, flash-updateable and/or the like.

Specialized computer system 1300 may additionally include acomputer-readable storage media reader 1350, a communications system1360 (e.g., a modem, a network card (wireless or wired), an infra-redcommunication device, Bluetooth™ device, cellular communication device,etc.), and working memory 1380, which may include RAM and ROM devices asdescribed above. In some embodiments, specialized computer system 1300may also include a processing acceleration unit 1370, which can includea digital signal processor, a special-purpose processor and/or the like.

Computer-readable storage media reader 1350 can further be connected toa computer-readable storage medium, together (and, optionally, incombination with storage device(s) 1340) comprehensively representingremote, local, fixed, and/or removable storage devices plus storagemedia for temporarily and/or more permanently containingcomputer-readable information. Communications system 1360 may permitdata to be exchanged with a network, system, computer and/or othercomponent described above.

Specialized computer system 1300 may also comprise software elements,shown as being currently located within a working memory 1380, includingan operating system 1384 and/or other code 1388. It should beappreciated that alternate embodiments of specialized computer system1300 may have numerous variations from that described above. Forexample, customized hardware might also be used and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets), or both. Furthermore, connection to othercomputing devices such as network input/output and data acquisitiondevices may also occur.

Software of specialized computer system 1300 may include code 1388 forimplementing any or all of the function of the various elements of thearchitecture as described herein. For example, software, stored onand/or executed by a specialized computer system such as specializedcomputer system 1300, can provide the functions of components of theinvention such as those discussed above. Methods implementable bysoftware on some of these components have been discussed above in moredetail.

List of Some Exemplary Embodiments

Embodiment 1: A method (1000) comprising: obtaining (1001) one or moreimages of a left eye and one or more images of a right eye; computing(1002), based on the one or more images of the left eye, gaze trackingdata for the left eye indicating an estimated gaze direction of the lefteye; computing (1003), based on the one or more images of the right eye,gaze tracking data for the right eye indicating an estimated gazedirection of the right eye; computing, based on the one or more imagesof the left eye, a plurality of parameters representing differentfactors indicative of a reliability of the gaze tracking data for theleft eye; and computing, based on the one or more images of the righteye, a plurality of parameters representing different factors indicativeof a reliability of the gaze tracking data for the right eye.

Embodiment 2: The method of embodiment 1, further comprising: computing(1004) a combined reliability parameter for the gaze tracking data forthe left eye based on the plurality of parameters computed for the lefteye; and computing (1005) a combined reliability parameter for the gazetracking data for the right eye based on the plurality of parameterscomputed for the right eye.

Embodiment 3: The method of embodiment 2, further comprising: computing(1006) a combined gaze direction or gaze point based on the gazetracking data for the left and right eyes and the associated combinedreliability parameters.

Embodiment 4: The method of embodiment 3, wherein computing the combinedgaze direction or gaze point includes: forming a weighted combination ofthe gaze tracking data for the left eye and the gaze tracking data forthe right eye, wherein the gaze tracking data for the respective eyesare weighted based on the respective associated combined reliabilityparameters.

Embodiment 5: The method of any of the preceding embodiments, whereincomputing the gaze tracking data for an eye (200) includes: estimating,based on the one or more images of the eye, a position of a pupil (210)of the eye and a position of a glint (220) at the eye; and computing,based on the estimated pupil position and glint position for the eye,the gaze tracking data for the eye.

Embodiment 6: The method of any of the preceding embodiments, whereinthe plurality of parameters computed for an eye (300) includes: aposition of a glint (220) at the eye relative to a cornea (310) of theeye; a distance between a glint (220) at the eye and a center (313) of acornea (310) of the eye; a position of a glint (220) at the eye relativeto a pupil (210) of the eye; a distance between a glint (220) at the eyeand a center (350) of a pupil of the eye; and/or a position of a glint(610) at the eye relative to an edge (620) of a pupil (630) of the eye.

Embodiment 7: The method of embodiment 6, comprising: computing (1004) acombined reliability parameter for the gaze tracking data for the lefteye based on the plurality of parameters computed for the left eye; andcomputing (1005) a combined reliability parameter for the gaze trackingdata for the right eye based on the plurality of parameters computed forthe right eye, wherein computing a combined reliability parameter forthe gaze tracking data of an eye comprises: associating a first value ofa distance between the glint at the eye and the center of the cornea ofthe eye with a lower reliability than a reliability associated with asecond value of said distance, wherein the first value of said distanceis higher than the second value of said distance; and/or associating asituation where the glint at the eye is located at the edge of the pupilof the eye with lower reliability than reliability associated with otherpositions of the glint relative to the pupil.

Embodiment 8: The method of any of the preceding embodiments, whereinthe plurality of parameters computed for an eye 510 includes: a contrastlevel between a pupil 511 of the eye and an iris of the eye.

Embodiment 9: The method of embodiment 8, comprising: computing (1004) acombined reliability parameter for the gaze tracking data for the lefteye based on the plurality of parameters computed for the left eye; andcomputing (1005) a combined reliability parameter for the gaze trackingdata for the right eye based on the plurality of parameters computed forthe right eye, wherein computing a combined reliability parameter forthe gaze tracking data of an eye comprises: associating a first value ofthe contrast level with a lower reliability than a reliabilityassociated with a second value of the contrast level, wherein the firstvalue of the contrast level is lower than the second value of thecontrast level.

Embodiment 10: The method of any of the preceding embodiments, whereinthe plurality of parameters computed for an eye (200) includes: anintensity of a glint (220) employed to compute the gaze tracking datafor the eye; a shape of a glint (220) employed to compute the gazetracking data for the eye; and/or a number of glints (220) detected inthe one or more images of the eye employed to obtain the gaze trackingdata for the eye.

Embodiment 11: The method of embodiment 10, comprising: computing (1004)a combined reliability parameter for the gaze tracking data for the lefteye based on the plurality of parameters computed for the left eye; andcomputing (1005) a combined reliability parameter for the gaze trackingdata for the right eye based on the plurality of parameters computed forthe right eye, wherein computing a combined reliability parameter forthe gaze tracking data of an eye comprises: evaluating a similaritybetween said shape and an expected glint shape; and associating a firstlevel of similarity between the shapes with a lower reliability than areliability associated with a second level of similarity between theshapes, wherein the first level or similarity is lower than the secondlevel of similarity.

Embodiment 12: The method of any of embodiments 10-11, comprising:computing (1004) a combined reliability parameter for the gaze trackingdata for the left eye based on the parameters computed for the left eye;and computing (1005) a combined reliability parameter for the gazetracking data for the right eye based on the parameters computed for theright eye, wherein computing a combined reliability parameter for thegaze tracking data of an eye comprises: comparing the number of glintsdetected in the one or more images of the eye employed to obtain thegaze tracking data for the eye with an expected number of glints; andassociating a situation where the detected number of glints coincideswith the expected number of glint with a higher reliability than areliability associated with a situation where the detected number ofglints deviates from the expected number of glints.

Embodiment 13: The method of any of the preceding embodiments, whereinthe plurality of parameters computed for an eye includes: a parameterindicating presence of a reflection from glasses in the one or moreimages of the eye; and/or a parameter indicating whether a certainregion of the eye is at least partially obscured in the one or moreimages of the eye.

Embodiment 14: The method of any of the preceding embodiments, whereinthe plurality of parameters computed for an eye (600) includes: aparameter indicating a number of pupil edge pixels in the one or moreimages of the eye, wherein the pupil edge pixels are image pixelslocated along those one or more portions of the edge (620) of the pupil(630) which are visible in the one or more images of the eye.

Embodiment 15: The method of embodiment 14, comprising: computing (1004)a combined reliability parameter for the gaze tracking data for the lefteye based on the plurality of parameters computed for the left eye; andcomputing (1005) a combined reliability parameter for the gaze trackingdata for the right eye based on the parameters computed for the righteye, wherein computing a combined reliability parameter for the gazetracking data of an eye comprises: associating a first number of pupiledge pixels with a lower reliability than a reliability associated witha second number of pupil edge pixels, wherein the first number of pupiledge pixels is lower than the second number of pupil edge pixels.

Embodiment 16: The method of any of the preceding embodiments, whereinthe plurality of parameters computed for an eye includes: a parameterindicating a magnitude of residual errors remaining after calibration ofan eye model employed for computing gaze tracking data for the eye.

Embodiment 17: The method of any of the preceding embodiments,comprising, for each of the left and right eyes: estimating, based onone or more images of the eye, a position of a first glint at the eyecaused by reflection of light from a first illuminator (111);predicting, based on the position of the first glint and based onknowledge of how a second illuminator (112) is arranged relative to thefirst illuminator, a position of a second glint at the eye caused byreflection of light from the second illuminator; and detecting in one ormore images of the eye, a position of the second glint at the eye causedby reflection of light from the second illuminator, wherein theplurality of parameters computed for an eye includes: a differencebetween the predicted position and the detected position of the secondglint.

Embodiment 18: The method of any of the preceding embodiments,comprising: performing gaze tracking for the left eye and the right eyeindividually for a sequence of images; monitoring variability of gazetracking positions estimated for the left eye; and monitoringvariability of gaze tracking positions estimated for the right eye,wherein the plurality of parameters computed for an eye includes: aparameter indicative of the monitored variability of the gaze trackingpositions estimated for the eye.

Embodiment 19: The method of any of the preceding embodiments,comprising: performing gaze tracking for the left eye for a sequence ofimages; estimating distances to the left eye for the sequence of imagesemployed for gaze tracking for the left eye; estimating a noise level inthe estimated distances to the left eye; performing gaze tracking forthe right eye for a sequence of images; estimating distances to theright eye for the sequence of images employed for gaze tracking of theright eye; and estimating a noise level in the estimated distances tothe right eye, wherein the plurality of parameters computed for an eyeincludes: a parameter indicative of the noise level in the estimateddistances to the eye.

Embodiment 20: The method of any of the preceding embodiments, whereinthe plurality of parameters computed for an eye includes: a parameterindicative of a distance between a reference position and an average ofgaze tracking positions estimated for the eye.

Embodiment 21: The method of any of the preceding embodiments, whereinthe plurality of parameters computed for an eye includes: a parameterindicating whether the eye is dominant over the other eye.

Embodiment 22: A computer program product comprising one or morecomputer-readable media with instructions for performing the method ofany of embodiments 1-21.

Embodiment 23: A system (100) comprising: one or more processors (120)configured to perform the method of any of embodiments 1-21.

Embodiment 24: The system of embodiment 23, further comprising: one ormore illuminators (111, 112) for illuminating the eyes; and one or morecameras (112) for capturing images of the eyes.

Embodiment 25: A (1100) method comprising: receiving (1101) gazetracking data indicating a gaze direction of a left eye; receiving(1102) gaze tracking data indicating a gaze direction of a right eye;receiving (1103) a plurality of parameters representing differentfactors indicative of a reliability of the gaze tracking data for theleft eye; receiving (1104) a plurality of parameters representingdifferent factors indicative of a reliability of the gaze tracking datafor the right eye; computing (1105) a combined reliability parameter forthe gaze tracking data for the left eye based on at least one of theparameters associated with the left eye; and computing (1106) a combinedreliability parameter for the gaze tracking data for the right eye basedon at least one of the parameters associated with the right eye.

Embodiment 26: The method of embodiment 25, further comprising:computing (1107) a combined gaze direction or gaze point based on thegaze tracking data for the left and right eyes and the associatedcombined reliability parameters.

Embodiment 27: The method of embodiment 26, wherein computing thecombined gaze direction or gaze point includes: forming a weightedcombination of the gaze tracking data for the left eye and the gazetracking data for the right eye, wherein the gaze tracking data for therespective eyes are weighted based on the respective associated combinedreliability parameters.

Embodiment 28: The method of any of embodiments 25-27, wherein computinga combined reliability parameter for the gaze tracking data for an eyebased on at least one of the received parameters associated with the eyecomprises: selecting one or more of the plurality of parametersassociated with the eye; and computing the combined reliabilityparameter for the gaze tracking data for the eye based on the selectedone or more parameters.

Embodiment 29: The method of any of embodiments 25-28, wherein theplurality of parameters representing different factors indicative of areliability of the gaze tracking data for the left eye and the righteye, respectively, include one or more of the parameters A-Q listed inTable 1.

Embodiment 30: A method (1200) comprising: receiving (1101) gazetracking data indicating a gaze direction and/or gaze point of a lefteye; receiving (1102) gaze tracking data indicating a gaze directionand/or gaze point of a right eye; receiving (1203) a combinedreliability parameter for the gaze tracking data for the left eye;receiving (1204) a combined reliability parameter for the gaze trackingdata for the right eye; and computing (1107) a combined gaze directionor gaze point based on the gaze tracking data for the left and righteyes and the associated combined reliability parameters.

Embodiment 31: The method of embodiment 30, wherein computing thecombined gaze direction or gaze point includes: forming a weightedcombination of the gaze tracking data for the left eye and the gazetracking data for the right eye, wherein the gaze tracking data for therespective eyes are weighted based on the respective associated combinedreliability parameters.

Embodiment 32: A computer program product comprising one or morecomputer-readable media with instructions for performing the method ofany of embodiments 25-31.

Embodiment 33: A system comprising: one or more processors configured toperform the method of any of embodiments 25-31.

What is claimed is:
 1. A system for determining a gaze direction of auser, wherein the system comprises: an eye tracking device configuredto: capture image data relating to images of a left eye of a user;capture image data relating to a right eye of the user; determine a gazedirection of the left eye of the user based on the image data relatingto the left eye; determine a gaze direction of the right eye of the userbased on the image data relating to the right eye; determine a firstreliability parameter based on a feature from the image data relating tothe left eye, the first reliability parameter representing a reliabilityof the gaze direction of the left eye; determine a second reliabilityparameter based on a feature from the image data relating to the righteye, the second reliability parameter representing a reliability of thegaze direction of the right eye; and determine a final gaze directionbased on the first reliability parameter and the second reliabilityparameter.
 2. The system for determining a gaze direction of a user ofclaim 1, wherein: determining the final gaze direction is further basedon a comparison of the gaze direction of the left eye with the gazedirection of the right eye.
 3. The system for determining a gazedirection of a user of claim 2, wherein: determining the final gazedirection is further based on a comparison of the first reliabilityparameter with the second reliability parameter.
 4. The system fordetermining a gaze direction of a user of claim 1, wherein: determiningthe first reliability parameter or determining the second reliabilityparameter, respectively, is further based on at least one of: a glintintensity at the left eye or the right eye, respectively; a shape ofglints at the left eye or the right eye, respectively; or a number ofglints at the left eye or the right eye, respectively.
 5. The system fordetermining a gaze direction of a user of claim 1, wherein: determiningthe first reliability parameter or determining the second reliabilityparameter is further based on a head pose angle of the user.
 6. Thesystem for determining a gaze direction of a user of claim 1, wherein:determining the final gaze direction is further based on an eyedominance parameter which indicates whether the left eye or the righteye is dominant.
 7. The system for determining a gaze direction of auser of claim 1, wherein: determining the first reliability parameter ordetermining the second reliability parameter, respectively, is furtherbased on a number of pupil edge pixels in the image data relating to theleft eye or the image data relating to the right eye, respectively,wherein a lower number of pupil edge pixels is associated with a lowerreliability parameter.
 8. A method for determining a gaze direction of auser, the method comprising: capturing image data relating to a left eyeof a user; capturing image data relating to a right eye of the user;determining a gaze direction of the left eye of the user based on theimage data relating to the left eye; determining a gaze direction of theright eye of the user based on image data relating to the right eye;determining a first reliability parameter based on a feature from theimage data relating to the left eye, the first reliability parameterrepresenting a reliability of the gaze direction of the left eye;determining a second reliability parameter based on a feature from theimage data relating to the right eye, the second reliability parameterrepresenting a reliability of the gaze direction of the right eye; anddetermining a final gaze direction based on the first reliabilityparameter and the second reliability parameter.
 9. The method fordetermining a gaze direction of a user of claim 8, wherein: determiningthe final gaze direction is further based on an eye dominance parameter.10. The method for determining a gaze direction of a user of claim 8,wherein: determining the first reliability parameter or the secondreliability parameter, respectively, is further based on: a position ofa glint relative to a cornea of the left eye or the right eye,respectively; a distance between the glint and a cornea center of theleft eye or right eye, respectively; a position of the glint relative toa pupil of the left eye or right eye, respectively; a distance betweenthe glint and a center of the pupil of the left eye or right eye,respectively; an intensity of the glint detected on the left eye or theright eye respectively; a contrast level between the pupil of the lefteye or the right eye, and an iris of the left eye or right eye,respectively; a position of the glint relative to an edge of the pupilof the left eye or the right eye, respectively; a parameter indicating anumber of image pixels located along an edge of the pupil of the lefteye or the right eye, respectively; a difference between a predictedglint position and a detected glint position on the left eye or theright eye, respectively; a shape of the glint on the left eye or theright eye, respectively; a number of glints detected on the left eye orthe right eye, respectively; or a parameter indicating a degree ofspatial distribution of the image pixels located along the edge of thepupil on the left eye or the right eye, respectively.
 11. The method fordetermining a gaze direction of a user of claim 8, wherein: determiningthe first reliability parameter or determining the second reliabilityparameter, respectively, is further based on: an indication of areflection in the left eye or the right eye, respectively; or anindication that a region of the left eye or the right eye, respectively,is at least partially obscured.
 12. The method for determining a gazedirection of a user of claim 8, wherein: determining the firstreliability parameter or the second reliability parameter, respectively,is further based on: a parameter indicating a precision of the gazetracking of the left eye or the right eye, respectively; an aggregationof gaze directions of the left eye or the right eye, respectively; anindication of a magnitude of residual errors following a calibration ofan eye model; or a confidence value estimated by a machine learningalgorithm trained based on comparing gaze direction data with knownobject positions.
 13. The method for determining a gaze direction of auser of claim 8, wherein: determining the first reliability parameter orthe second reliability parameter, respectively, is further based on: anaverage distance between a user and an intersection of a gaze directionwith a physical object; or a noise level in an estimated distance fromthe user to an object intersected by the gaze direction.
 14. The methodfor determining a gaze direction of a user of claim 8, wherein:determining a gaze direction of the left eye of the user is furtherbased on detecting a glint in the image data relating to the left eye inresponse to selectively illuminating the left eye.
 15. A non-transitorycomputer readable medium having instructions stored thereon, fordetermining a gaze direction of a user, executable by a computing deviceto cause the computing device to perform operations comprising:capturing image data relating to a left eye of a user capturing imagedata relating to a right eye of the user; determining a gaze directionof the left eye of the user based on the image data relating to the lefteye; determining a gaze direction of the right eye of the user based onthe image data relating to the right eye; determining a firstreliability parameter based on a feature from the image data relating tothe left eye, the first reliability parameter representing a reliabilityof the gaze direction of the left eye; determining a second reliabilityparameter based on a feature from the image data relating to the righteye, the second reliability parameter representing a reliability of thegaze direction of the right eye; and determining a final gaze directionbased on the first reliability parameter and the second reliabilityparameter.
 16. The non-transitory computer readable medium of claim 15,wherein: determining the gaze direction of the left eye is based furtheron detecting a glint in the image data relating to the left eye inresponse to selectively illuminating the left eye.
 17. Thenon-transitory computer readable medium of claim 15, wherein:determining the final gaze direction is further based on an eyedominance parameter which indicates whether the left eye or the righteye is dominant.
 18. The non-transitory computer readable medium ofclaim 15 wherein the operations further comprise: detecting a firstglint caused by a first selective illuminator; predicting a location ofa predicted glint caused by a second selective illuminator based on thefirst glint; and detecting a second glint caused by the second selectiveilluminator and wherein determining the first reliability parameter orthe second reliability parameter, respectively, is based on a comparisonof the predicted glint with the second glint.
 19. The non-transitorycomputer readable medium of claim 15, wherein: determining the finalgaze direction is further based on a comparison of the gaze direction ofthe left eye with the gaze direction of the right eye.
 20. Thenon-transitory computer readable medium of claim 15, wherein:determining the final gaze direction is further based on a comparison ofthe first reliability parameter with the second reliability parameter.